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ABSTRACT 


Simultaneous in-class presentations are well suited to the use of peer evaluation, which also promotes greater 
involvement of the student audience. The problem for the teacher is how to manage peer evaluation and make it a useful 
part of the curriculum. PeerEval is a mobile application that allows students to anonymously evaluate presentations in 
real time using a Likert scale rubric and individual peer comments. The results of each evaluation are compiled in a 
database which is available to the teacher and the students. This short paper focuses on implementing and evaluating this 
app in Japanese university classrooms. The researchers sought to evaluate both the technical aspects of the software and 
nature of student feedback using the software. Student attitudes towards the PeerEval system were measured using a 
twelve-item questionnaire concerning usability of the software, their attitudes towards the system both as a presenter and 
as an audience member. Results are discussed regarding student perceptions of the evaluation system, overall feedback 
quality, and the perceived effect of feedback speed and peer comments. Further uses for a mobile peer-evaluation system 
are also discussed. 
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1. INTRODUCTION 


PeerEval (https://peereval.mobi/) is a peer evaluation application developed to be used with simultaneous 
in-class presentations. These kinds of presentations have a number of positive benefits, such as reducing 
speaking anxiety and promoting greater involvement of the students in the presentations (Cote, 2013). 
PeerEval seeks to deal with a number of drawbacks to current forms of peer evaluation such as, considerable 
paperwork, a reluctance to fairly evaluate peers or comment critically, and asynchronous rating. 

This short paper gives an explanation of the development of the PeerEval system, a description of the use 
of the app, and a user evaluation of the software. Student attitudes towards PeerEval were measured using a 
twelve-item questionnaire concerning usability of the software and their attitudes towards the system as a 
presenter and as an audience member. 


2. PEER EVALUATION OF PRESENTATIONS 


Modern teaching practices such as poster presentations have allowed for more active learning by giving 
students more performance time in language classrooms. One of the drawbacks with such activities is that the 
teacher must, by the very nature of the activity, play a smaller role. It is almost impossible for the teacher to 
evaluate all students in such an activity. This situation leads to the implementation of some sort of peer 
evaluation. With proper planning and careful execution, peer evaluation can have a positive effect on both 
performance and L2 acquisition (Hansen & Liu, 2005; Gobel & Kano, 2017). 

Peer assessment has been used as an alternative evaluation method for a variety of oral presentation 
activities (Boud, Cohen & Sampson, 1999; Patri, 2002; Shimura, 2006). It can encourage active involvement 
of the students, foster collaborative learning, and contribute to learner autonomy (Donato, 1994; Kessler 
& Bikowski, 2010; Rust, Price, & O’Donovan, 2003; Tseng & Tsai, 2010). 
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With the above benefits in mind, many teachers and textbooks have created paper-based rubrics for peer 
evaluation. While these paper-based assessment forms are useful, they pose a number of problems. One 
problem is the issue of anonymity. Without some privacy and anonymity, the effect of social influence may 
increase (Panadero, Romero, & Strijbos, 2013; Tseng, et al., 2010). In addition to this, collating paper 
evaluations is a time-consuming task. 

Some of these problems can be alleviated by using LMS such as Moodle or Blackboard, which make it 
easy for students to input scores online and for the teacher to manage these scores. Such systems have been 
used to great effect in the peer assessment of writing (see Davies, 2000, for an overview). But it is still 
difficult to ensure total anonymity of feedback and comments on a forum. Some research has been done with 
classroom response technologies (CRT) and anonymity (Bojinova and Ojigara, 2011, 2013; Raes, 
Vanderhoven., & Schellens, 2013), but these dealt with simple assessment scores, as opposed to written 
feedback. Also, CRT assessment is usually done asynchronously. 

To solve these problems, PeerEval was created with a simple interface that is both flexible and 
anonymous in nature. The application sought to create evaluation rubrics that could be used quickly and 
anonymously to give quick and accurate impressions to their peers, as well as giving users immediate access 
to their scores and feedback. The following section describes the PeerEval system and how it has been used 
for in-class presentations and peer assessment. 


2.1 Description of the System 


There are two components: a browser-based system for the instructor to create the evaluation criteria, to 
upload the student name list, and to download the results; and the iOS app that the students use for their 
assessments. In addition, from 2018 students can access the PeerEval student system (the same as the app) on 
any web browser. 

Teachers need to go to http://peereval.mobi, where there are two choices. Teachers may use the system 
without registration, but they must configure their session, conduct their class and download their results 
within a set time period. Teachers with login-access can create multiple rubrics, sessions and classes that 
remain in the system until the teacher elects to delete them. 

Teachers create a session by naming the session, inputting student information and choosing or creating a 
rubric. Student information can be input on the webpage itself or by uploading a .csv file with all the relevant 
information. 

Within PeerEval, teachers can choose one of the default rubrics or create their own. A set of up to six 
rubrics can be set up for one session, with a choice of four-point or five-point Likert scales. Each rubric 
consists of a title for the item, a description of the item, and a 4-5 point Likert scale (see Figure 1). 
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Figure 1. Rubric set up 


Once a session has been fully set up, an access code is created which will be used by the students, along 
with their handle to access the session via the app. 

Once a session has been completed, the teacher can view or download the results. The teacher can show 
the results instantly on the class screen if s/he is not concerned about student privacy, or print them out and 
supply the students with their own scores, which also show the class averages (Figure 2). 
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Student Scores 


Figure 2. Individual score report 


2.1.1 Student Attitudes towards the PeerEval App 


To informally measure student attitudes towards PeerEval, a twelve-item questionnaire was created 
addressing affective, practical, and technical issues (Chen, 2014), and administered to students who had 
experience with both paper-based and PeerEval peer evaluation procedures. The rubrics for both procedures 
were similar. The questionnaire was divided into five questions about the PeerEval app itself, three questions 
comparing paper-based and app-based evaluation from the presenter’s point of view, and four questions 
comparing paper and app-based evaluation from the point of view of the audience. The questionnaire was 
administered to a total of 39 students, enrolled in four separate classes. In all four classes, the students first 
gave short presentations and were graded using a paper-based rubric. In the following month, they gave a 
separate presentation and were evaluated on a similar rubric via PeerEval. Cronbach’s alpha for the results 
was .73, indicating acceptable but weak internal consistency. 

Table | shows the result of the questionnaire. As for Items 1-5, a 4-point Likert scale was used, | being 
“Agree,” and 5 being “Don’t agree.” As for Items 6-12, a 4-point Likert scale was used, | being the strong 
preference to paper-based evaluation, and 4 being strong preference to the app-based evaluation. In general, 
students felt that the app was easy to install and use (Items 1-5). They felt that the evaluation system was 
clear and that the comments and feedback would help them prepare more for subsequent presentations. From 
the presenters’ standpoint, the feedback speed was a major plus, and the ability to quickly read peer 
comments was viewed as helpful (Items 6-7). From the point of view of the audience, everyone preferred the 
privacy and feedback speed of the app and felt that the app gave them the feeling that the presentations were 
more interactive (Item 12). The students also felt that the app allowed them to evaluate more accurately than 
using paper-based methods (Items 9-11). 


Table 1. Questionnaire Results 


The preliminary results of comparing PeerEval with paper-based forms of peer evaluation allowed us to 
consider student preferences. Students generally had a favorable impression of the system, but presenters felt 
audience members were paying less attention to presentations and more attention to their phones. This 
preliminary result suggests that rules and/or procedures should be put into place regarding when students are 
allowed to input their evaluations. 


3. CONCLUSION 


This paper briefly described a peer assessment mobile app, how it is used in context, and student attitudes 
towards the use of the app in reference to a paper-based rubric. The results of this preliminary study suggest 
that PeerEval-generated interaction quality and quantity may be different from asynchronous peer feedback. 

Although PeerEval was designed to overcome rating problems with paper and LMS-based forms of peer 
evaluation, the present study only compared students’ perceptions of paper-based and PeerEval peer 
evaluation procedures, without collecting data about LMS-based peer evaluation. In addition, previous 
studies have shown a _ positive effect of training in technology-based peer evaluation 
(Ho & Savignon, 2007) suggesting that more detailed feedback training will produce more useful, helpful, 
and relevant comments. 
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